Aiming at detecting chromatic moving object from video sequences with fixed scene．HSV color space was adopted to represent each raw frame in video sequences，then it established background frame based on self-adaptive background model by extracting V value in raw frame, and detected moving object based on background subtraction．Motion region binarization incorporated H value in raw frame to filter motion region，in order to obtain the foreground frame in which the special moving object was selected．The experimental results show that the algorithm can achieve the special color moving object detection in the video with fixed scene and several moving objects very well．